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ABSTRACT 

Lighting  design  plays  an  important  role  in  interactive  3-D 
training  environments;  it  influences  scene  perception, 
scene  understanding,  attention,  visual  search,  presence, 
immersion,  and  emotional  involvement.  Although  there 
are  several  research  projects  that  measured  the  impact  of 
lighting  in  3-D  environments,  little  research  focused  on 
the  development  of  new  lighting  design  models  to 
enhance  the  interactive  experience.  Current  lighting 
design  techniques  rely  on  static  manual  designs.  Since 
interactive  training  environments  are  dynamic  and 
unpredictable,  such  static  designs  tend  to  be  inflexible  and 
do  not  adequately  support  the  interaction.  In  this  paper,  I 
will  discuss  examples  showing  problems  with  current 
lighting  design  approaches.  In  addition,  I  will  discuss 
ELE,  an  Expressive  Lighting  Engine,  that  automatically 
adapts  the  lighting  to  the  interaction  while 
accommodating  scenario  goals,  user  perception,  attention, 
immersion,  and  emotional  involvement.  I  also  discuss  the 
implication  and  the  use  of  ELE  in  interactive  3-D  training 
environments. 


1.  INTRODUCTION 

Interactive  3-D  environments  are  becoming 
increasingly  important  for  their  demonstrated  utility  in 
military  training  environments  (ABCNews,  august  21, 
2003;  Sieberg,  November  23,  2001;  Stanney,  2002; 
Swartout  et  al.,  2003;  Wired,  2003).  The  wide  deployment 
of  game  engines,  such  as  Unreal  Tournament 
technologies,  have  facilitated  and  accelerated  the 
development  process  of  such  immersive  training 
applications.  Even  with  these  tools,  however,  creating 
training  environments  still  require  much  time  and  effort. 
In  addition,  these  engines  often  do  not  include  methods 
for  adapting  to  or  accounting  for  the  user’s  perceptual  and 
cognitive  processes  during  training.  There  are  several 
limitations  with  current  tools,  especially  in  their  use  for 
training  environments.  In  this  paper,  I  specifically  focus 
on  lighting  design. 


Lighting  design  plays  a  very  important  role  in 
creating  immersive  3-D  environments.  It  influences  user’s 
perception,  scene  understanding,  attention,  visual  search 
process,  presence,  and  emotional  involvement.  It  should 
be  noted  that  lighting  is  not  the  only  factor  that  affects 
these  perceptual  and  cognitive  processes;  it  is  among 
several  visual  and  audio  components  that  are  used  for  this 
purpose,  including  ambient  sounds,  camera,  and  textures. 

Lighting  has  been  used  by  many  film,  theatre,  and 
animation  designers  to  set  atmosphere,  evoke  moods,  and 
heighten  the  emotional  engagement.  These  qualities  are 
very  important  for  3-D  training  environments.  Research 
has  shown  that  in  emotionally  heightened  situations 
memory  retention  increases  (Ulate,  2002).  Thus,  the  use 
of  lighting  to  increase  emotional  engagement  can 
stimulate  better  memory  retention.  Lighting  has  also  been 
used  by  many  designers  to  provide  and  retain  attention 
and  involvement  (Alton,  1995;  Birn,  2000;  Brazel,  1997; 
Calahan,  1996).  This  is  specifically  important  for  training 
environments,  since  continuous  concentration  and 
attention  is  needed  to  complete  training  scenarios  and 
motivate  learning. 

To  develop  a  lighting  design  for  interactive  3-D 
environments,  current  designers  predefine  the  number  of 
lights  used  for  each  set,  and  predefine  their  positions, 
angles,  and  colors  depending  on  the  desired  atmosphere, 
the  light  sources  present  in  the  set,  the  time  of  day,  and 
the  location.  This  static  manual  design  is  inflexible. 
Physical  and  scenario  configurations  of  interactive  3-D 
environments  often  change  unpredictably  due  to  user 
interaction.  Thus,  a  lighting  design  that  is  configured  for  a 
particular  mood  or  atmosphere  may  no  longer  be  valid 
given  the  run-time  fluctuations  of  the  user’s  emotional 
engagement  and  the  situation  given  user’s  interaction. 
Also,  a  lighting  design  that  is  configured  for  guiding 
trainees’  visual  search  process  inherently  does  not 
account  for  the  dynamic  change  of  focus  depending  on 
the  trainee’s  performance  and  scenario  goals. 

To  better  facilitate  training  goals  and  enhance  current 
training  environments,  I  propose  the  use  of  an  automatic 
lighting  design  technique  that  adapts  the  lighting 
accommodating  the  interaction  and  scenario  goals  as  well 
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as  accounting  for  lighting  design  functions,  including 
attention,  emotional  stimulation,  and  enhancing  scene 
understanding  and  perception.  In  this  paper,  I  discuss  an 
automatic  lighting  design  system,  called  ELE  (Expressive 
Lighting  Engine).  In  particular,  I  present  results  and 
discuss  implications  and  utility  of  ELE  in  training 
simulations. 


2.  FACTORS  INFLUENCED  BY  LIGHTING 
IN  3-D  ENVIRONMENTS 

2.1  Realistic  appearance  and  Rendering 

Many  training  environments  have  emphasized 
realism.  Although  realism  may  be  important  for  many 
training  applications,  it  is  not  essential  for  achieving 
presence,  attention,  and  emotional  engagement 
(Zimmons,  2004).  Also,  realism  is  not  always  in  conflict 
with  providing  atmosphere  or  mood,  as  one  may  believe. 
Many  film-based  techniques  have  been  employed  by 
cinematographers,  where  subtle  changes  in  contrast  and 
exposure  levels  are  applied  to  the  scene  to  heighten  the 
audience’s  emotional  experience,  while  maintaining  the 
perceived  realism  of  the  environment  portrayed  (Block, 
2001;  Brown,  1996;  Viera,  1993).  These  methods  elevate 
tension,  anticipation,  and  increase  attention,  emotional 
engagement,  and  presence. 

Achieving  realistic  lighting  in  real-time  is  still  a 
challenge.  Real-time  rendering  techniques  used  by  many 
game  engines  and  interactive  3-D  environments  often 
approximate  the  lighting  process,  e.g.  they  do  not  account 
for  refractions  or  inter-reflections  of  light  (Moller  & 
Haines,  1999).  This  has  forced  designers  to  adopt  several 
tricks  to  produce  more  realistic  rendering  effects,  e.g. 
designers  use  ambient  lights  to  simulate  the  reflections  of 
light  on  the  different  surfaces  (Birn,  2000).  Experienced 
designers  use  these  tricks  and  carefully  balance  their  use, 
knowing  that  too  much  ambient  light  tend  to  flatten  the 
image  (Birn,  2000)  and  no  ambient  light  leads  to  darker 
and  less  realistic  scenes. 

Global  illumination  is  a  term  used  by  the  graphics 
community  to  refer  to  the  general  problem  of  calculating 
light  propagation  in  simulated  environments,  accounting 
for  the  many  inter-reflections  between  scene  surfaces.  It 
has  been  shown  that  such  realistic  measures  are  not 
important  for  achieving  presence,  attention,  and  emotional 
engagement  (Zimmons,  2004).  Zimmons  conducted 
several  experiments  to  compare  rendering  algorithms 
(local  lighting  vs.  global  illumination)  and  the  impact  of 
using  a  more  realistic  rendering  algorithm  on  presence 
(Zimmons,  2004).  His  results  show  that  global 
illumination  didn’t  produce  significantly  better  presence 
than  a  less  realistic  illumination  algorithm.  This  concludes 
that  presence  and  attention  can  be  stimulated  without 


emphasis  on  realism.  Thus,  training  environments  that 
strives  for  ensuring  presence  or  emotional  engagement  do 
not  necessarily  need  realistic  rendering. 

2.2  Scene  Perception  and  Understanding 

Since  lighting  affects  scene  rendering,  it  inherently 
affects  users’  perception  and  scene  understanding.  This  is 
especially  important  for  fast  paced  environments,  where 
the  scenario  depends  on  the  quickness  of  the  trainee  in 
deciphering  the  environment  and  responding.  To  achieve 
better  scene  understanding  and  perception,  lighting 
designers  identified  several  goals  for  their  designs.  These 
goals  include:  achieving  good  visibility  of  important  areas 
and  landmarks,  provide  a  sense  of  visual  depth  or  depth  of 
field,  direct  viewer’s  attention  to  important  areas,  and 
configure  lights  to  match  the  local  light  sources  present, 
e.g.  torches  or  sunlight  through  a  window  (Brown,  1996; 
Millerson,  1991).  Designing  scene  lighting  to  satisfy  these 
goals  is  important  to  achieve  better  scene  perception  and 
understanding  for  interactive  environments  with  complex 
elaborate  scenes. 

2.3  Presence 

Presence  is  the  sense  of  being  teleported  into  a  virtual 
space  (Freeman,  Ijsselsteijn,  &  Lessiter,  2000;  Riva, 
Davide,  &  Ijsselsteijn,  2003;  Singer  &  Witmer,  1999). 
Many  elements  affect  presence,  e.g.  ambient  sound. 
Lighting  has  a  significant  impact  in  simulating  space 
(Alton,  1995;  Calahan,  1996;  Carson,  2000).  Lighting 
designers  identified  two  goals  that  have  direct  effect  on 
simulating  space:  (1)  the  use  of  local  light  sources  present 
in  the  scene  to  guide  the  process  of  configuring  light 
positions,  angles,  and  color,  and  (2)  the  setting  of  the 
lighting  atmosphere  by  choosing  and  balancing  colors  of 
light  to  set  a  particular  time  of  day,  and  feel  for  the 
environment  (Alton,  1995;  Block,  2001;  Brown,  1996; 
Calahan,  1996;  Gillette,  1998;  Millerson,  1991;  Palmer, 
1985). 

2.4  Emotional  Involvement 

Lighting  designers  have  identified  mood  and 
emotions  as  one  of  the  main  functions  or  requirements  of 
a  lighting  design.  They  modulate  the  lighting  color 
contrast  and  affinity  in  terms  of  saturation,  brightness,  and 
warmth/cool  colors  during  the  scene  and  between  scenes 
to  increase  tension  and  support  the  scenario  goals  (Block 
2001).  This  intensifies  the  action  and  heightens  the  scene 
presented,  which  in  turn  increases  the  emotional 
involvement.  It  should  be  noted  that  cinematographers 
often  use  visual  patterns  integrating  camera,  lighting,  and 
character  actions  together  to  heighten  the  emotional 
engagement  (Alton,  1995;  Block,  2001;  Campbell,  1999; 
Cheshire  &  Knopf,  1979). 
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3.  LIMITATIONS  OF  CURRENT  TECHNIQUES 

Since  there  are  no  lighting  design  models  specifically 
formulated  for  interactive  3-D  environments,  current 
models  adopt  a  cinematic  approach.  The  problem  with 
using  a  cinematic  approach  lies  in  the  difference  between 
film  and  real-time  interactive  environments.  In  designing 
a  movie,  the  designers  already  know  the  plot,  the  actions, 
the  desired  or  intended  emotions  and  moods,  and  in  some 
cases,  he/she  have  already  seen  rehearsals  and  talked  with 
producers  and  directors.  Interactive  3-D  environments,  on 
the  other  hand,  are  unpredictable  due  to  user  interaction. 
Hence,  design  parameters,  including  character/object 
locations  and  importance,  are  not  known  at  design  time. 
For  example,  character  relationships  change  depending  on 
user’s  behaviors  and  actions,  and  physical  positions  of 
characters  change  relative  to  the  user.  Therefore, 
designers  cannot  script  camera  actions,  lighting,  and 
staging  to  accommodate  all  situations  and  perform  desired 
perceptual  effects.  Thus,  current  manual  designs  often 
results  in  unwanted  or  undesirable  effects. 

In  addition,  this  method  forces  designers  to  explore  a 
large  space  of  possible  solutions  resulting  from  the 
combinations  of  low-level  parameters  and  possible 
scenarios.  While  experience  can  narrow  the  search  space, 
an  experienced  lighting  designer  still  requires  much  time 
and  effort  to  develop  an  adequate  design.  Most  often  such 
design,  because  of  its  static  nature,  does  not  adapt  to  the 
unpredictability  of  interactive  environment.  Thus,  most 
often  the  technique  leads  to  static  and  context-insensitive 
lighting  that  does  not  accommodate  variations  in  scene’s 
spatial  configuration,  mood,  or  visual  focus.  Examples  of 
such  problems  surface  in  many  interactive  productions, 
such  as  games  and  training  environments. 

3.1  No  dynamic  adaptation  of  the  design  to 

accommodate  interaction 

Figure  1  shows  a  screenshot  from  a  game  called 
Blade  of  Darkness  where  the  designer  aimed  to  stimulate 
mood  by  attaching  light  sources  to  torches  (practical 
sources1)  in  the  scene,  and  dynamically  calculating  the 
lighting  accordingly.  While  this  technique  produces 
effective  mood,  it  has  several  drawbacks.  Lighting  is 
solely  dependant  on  practical  sources  in  the  scene;  there  is 
no  method  for  manipulating  the  design  to  achieve  other 
visual  design  goals  such  as  visual  focus,  visibility,  visual 
intensity,  and  mood.  Therefore,  the  design  often  leads  to 
many  inappropriate  effects,  as  illustrated  in  figure  1 .  The 
figure  shows  insufficiently  lit  characters  or  action  due  to 
unpredicted  movements  and  the  absence  of  an  algorithm 
that  adapts  the  design  to  accommodate  changes  in  the 
environment.  In  figure  1,  the  participant  (whose  back  is 


1  Practical  sources  are  sources  suggested  by  the  set,  such 
as  lanterns,  windows,  or  torches. 


towards  us)  is  fighting  another  character  (facing  us)  who 
is  barely  visible,  and  in  this  case  the  participant  cannot 
determine  if  his/her  sword  touched  the  other  character  or 
not,  which  frustrates  him/her  and  diminishes  his/her 
engagement.  This  problem  surfaces  in  many  other  3-D 
applications  as  well. 


Fig.  1.  A  Screenshot  from  Blade  of  Darkness 


3.2  No  dynamic  light  modulation  for  tension 

Movies  and  games  all  use  techniques  to  intensify  the 
dramatic  moments  in  the  scene  (Alton,  1995;  Bim,  2000; 
Block,  2001;  Calahan,  1996;  Campbell,  1999;  Cheshire  & 
Knopf,  1979;  Maattaa,  2002).  Currently  game  designers 
have  developed  several  strategies  to  dynamically  intensify 
the  experience  based  on  audio,  camera  movement 
(Adventure,  2003),  texturing,  and  level  design  (Carson, 
2000;  Maattaa,  2002).  Most  of  these  methods  are 
inherently  tied  to  the  specific  game  or  application  the 
method  was  developed  for,  i.e.  they  are  not  generic 
enough  to  be  used  by  other  games  or  applications. 

While  cinematographers,  animators,  and  theatre  designers 
use  lighting  to  parallel  the  tension  in  the  scene  and  evoke 
emotions,  game  designers  minimally  use  lighting  for  this 
function.  Recently,  however,  many  techniques  have 
surfaced  showing  the  impact  of  using  dynamic  lights  for 
establishing  and  modulating  tension,  e.g.  Doom  3  (Macek, 
2004).  This  technique  is  still  not  adopted  in  all  games  due 
to  the  design  tools  and  the  established  production 
pipeline.  As  shown  by  recently  adopted  designs,  dynamic 
automatic  lighting  modulation  presents  a  potentially  less 
obtrusive  and  more  subtle  but  effective  solution  that  has 
yet  to  make  its  way  to  the  production  pipeline  of 
interactive  3-D  applications. 

Dynamic  lighting  techniques  currently  used  by  games, 
such  as  Doom  3,  are  still  in  early  development.  They 
currently  rely  on  fixed  predefined  patterns  that  designers 
set  and  trigger  depending  on  a  predefined  situation,  e.g. 
before  monsters  attach  or  when  user  enters  zone  x. 


3 


3.3  Problems  with  scene  understanding  and  visual 

attention 

We  conducted  an  experiment  to  measure  visual 
attention  and  scene  understanding  within  3-D 
environments.  We  asked  several  students  to  play  a 
popular  first  person  shooter  game  called  Unreal 
tournament  2003.  We  asked  them  to  wear  an  eye  tracker. 
We  recorded  their  eye  movements  superimposed  on  the 
game  video  for  analysis. 

From  our  observations  and  taped  interaction,  we 
deduced  that  the  number  of  misses  of  novice  users  is 
significantly  higher  than  that  of  expert  users.  These 
misses  have  caused  naive  users  to  be  killed  instantly.  One 
interaction  proceeded  as  follows.  A  user  enters  a  room, 
scans  the  environment  for  any  visible  enemies.  Even 
though  enemies  exist,  the  user  doesn’t  see  them.  The  user 
advances  into  the  room,  hears  gun  shots,  starts  shooting 
everywhere  while  scanning  the  environment  again, 
spotting  nothing.  The  user  looks  at  his  health,  notices  that 
his  health  is  dropping.  He  starts  scanning  the  environment 
again,  but  by  the  time  he  actually  sees  the  enemy,  he  was 
dead. 

These  results  are  significant  for  training  applications. 
The  results  show  that  in  complex  3-D  environments  naive 
users  were  not  able  to  spot  important  characters,  and  thus 
failed  to  respond.  This  shows  that  the  visual  search 
process  as  well  as  task  performance  was  influenced  by 
scene  composition  and  lighting  (later  I  will  show 
enhancements  of  lighting  that  alleviates  this  problem). 

Many  games  solve  this  problem  by  adding  a  halo 
around  characters  and  important  objects,  as  evident  in 
Unreal  Tournament  2004.  While  this  presents  a  workable 
solution,  it  contradicts  with  the  realistic  requirement 
established  for  lighting  design,  and  can  sometimes  be 
infeasible  for  applications  where  object  importance 
change  over  time. 


4.  ELE  -  EXPRESSIVE  LIGHTING  ENGINE 

To  alleviate  the  problems  discussed  above,  I 
developed  a  new  lighting  design  model  called  ELE 
(Expressive  Lighting  Engine)  that  automatically  and 
dynamically  adjust  the  lighting  in  an  interactive  scene 
accommodating  the  lighting  design  functions  (attention, 
emotional  involvement,  depth,  and  enhancing  scene 
understanding  and  perception)  given  the  situation  and 
user  actions  evaluated  at  run-time.  Since  ELE  has  been 
published  elsewhere  (Magy  Seif  El-Nasr  &  Horswill, 
2003;  Magy  Seif  El-Nasr  &  Horswill,  2004),  I  will  only 
summarize  ELE  in  this  section  to  give  the  reader  a  quick 
overview  of  the  inner  details  of  ELE.  For  a  more 
comprehensive  treatment  of  ELE,  readers  are  referred  to 


(Magy  Seif  El-Nasr  &  Horswill,  2003;  Magy  Seif  El- 
Nasr  &  Horswill,  2004). 

Based  on  the  lighting  functions  or  scene  factors 
influenced  by  lighting  discussed  above,  I  identified 
several  lighting  design  goals,  as  follows: 

•  Provide  necessary  visibility  of  action 

•  Maintain  illusion  of  light  source  (to  achieve 
realism) 

•  Provide  visual  focus 

•  Provide  depth 

•  Evoke  emotions 

•  Set  atmosphere 

•  Maintain  visual  continuity 

ELE  is  based  on  cinematic  and  theatrical  lighting 
design  theories  (Alton,  1995;  Block,  2001;  Brown,  1996; 
Crowther,  1989;  Gillette,  1998;  Millerson,  1991);  it  is 
designed  to  automatically  select  the  number  of  lights, 
their  positions,  colors,  and  angles  to  achieve  the  goals 
enumerated  above.  To  accomplish  this  task,  ELE  uses 
lighting  design  rules  represented  mathematically  in  an 
optimization  function.  The  use  of  optimization  is 
important  to  balance  conflicting  lighting  design  goals. 
While  adjusting  the  lighting,  ELE  maintains  visual 
continuity. 

I  assume  that  there  exists  a  system  that  passes  several 
parameters  to  ELE,  including  a  set  of  parameters 
describing  style,  local  light  sources,  scene  graph, 
characters’  dimensions,  dramatic  focus  (the 
area/characters  to  which  attention  should  be  directed), 
current  scenario  goals,  current  user  actions,  and  the 
dramatic  intensity  of  the  situation.  Using  these 
parameters,  ELE  computes  the  number  of  lights  to  be 
used.  For  each  of  these  lights,  it  computes  the  type  of 
instrument  (e.g.,  spot  light  or  point  light),  color  in  RGB 
color  space,  attenuation,  position  as  a  3D  point, 
orientation  including  the  facing  and  up  vectors,  range, 
masking  parameters,  and,  depending  on  the  light 
instrument  used,  the  Penumbra  and  Umbra  angles.  These 
parameters  are  given  to  a  rendering  engine  to  render  the 
frame. 

ELE  first  determines  where  to  direct  viewers’ 
attention  given  the  number  of  characters  in  the  frame  and 
the  dramatic  importance  of  their  actions.  I  use  the  term 
dramatic  focus  to  denote  the  area  where  attention  should 
be  directed.  ELE  then  dynamically  allocates  lights  to 
visible  areas  in  the  scene.  Once  lights  are  allocated  to 
areas,  ELE  selects  angles  and  colors  for  each  light  in  the 
scene,  thus  forming  a  light  setup.  The  light  setup  is  then 
given  to  the  rendering  engine  to  render  the  frame. 

ELE  divides  the  visible  area  into  n  different  areas.  It 
categorizes  these  areas  as:  focus,  describes  the  focus  of 
the  shot,  non-focus,  areas  surrounding  the  focus  area,  and 
background  areas.  ELE  assigns  lights  to  each  area 
depending  on  its  category.  The  angle  and  color  systems 
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assign  angles  and  colors  depending  on  the  number  of 
lights  assigned  to  the  area  and  its  category. 

4.1  Selecting  Angles 

The  angle  system  selects  an  angle  for  each  key2  light 
according  to  the  lighting  design  goals.  Cinematic  rules 
used  to  satisfy  these  goals  often  contradict  with  one 
another.  Angles  used  to  establish  mood,  for  example, 
don’t  usually  produce  good  visibility,  e.g.  rim  or 
silhouette  angles.  Thus,  I  softened  these  rules  into  cost 
functions,  where  the  designer  controls  weights  associated 
with  the  contradicting  requirements,  such  as  mood, 
visibility,  and  modeling: 

cost(&,  k~ ,  m)  =  A  V(k)  +  A _  I k  -  k~  I  +  A  I k  -  ml  + 

v  -I  I  -I  (1.1) 

Al  min  |  k  -l.  | , 

where  k  is  the  key  light  angle,  k~  is  the  key  light  angle 
from  the  previous  frame,  and  m  is  the  mood  angle 
suggested  by  the  artist,  Ay  is  the  cost  of  deviation  from  an 
orientation  of  light,  which  establishes  best  visibility,  A.  is 
the  cost  of  changing  the  key  light  angle  over  time  (to 
enforce  visual  continuity),  A /  is  the  cost  of  deviation  from 
an  established  light  source  angle/direction.  In  addition, 
for  each  light  source,  and  lt  is  the  angle  of  light  from  the 
light  source  i  on  the  subject  or  area  in  question,  and  Am  is 
the  cost  of  deviation  from  an  angle  that  shows  a  specific 
mood.  Based  on  Millerson’s  work  (Millerson,  1991),  I 
derived  the  following  formula  to  evaluate  visibility  and 
modeling: 

V  ( k )  =  sin(|  k  -  c|)  cos(&  -  s),  (1.2) 

where  k  and  c  are  the  azimuth  angles  of  the  key  light  and 
camera  relative  to  the  subject  and  s  is  the  azimuth  angle 
toward  which  the  subject  is  facing.  Millerson 
recommended  an  elevation  angle  between  ti/6  and  ti/3 
(Millerson,  1991). 

I  used  a  linear  optimization  algorithm  based  on  hill 
climbing  to  select  an  angle  for  each  key  light  that 
minimizes  the  cost  function  above. 

Fill  and  backlight  azimuth  angles  are  calculated 
depending  on  the  value  of  the  key  light  angle  and  the 
angle  between  the  camera  and  the  subject.  According  to 
the  guidelines  described  by  Millerson  (Millerson,  1991), 
fill  light  azimuth  and  elevation  angles  are  calculated  to  be 
the  mirror  image  of  the  key  light  angle.  I  derived  a 
formula  to  compute  backlight  azimuth  angle  as  follows: 

b  =  (&-c  +  ;r)mod2;z-,  (1.3) 

where  k  and  c,  again,  are  the  are  the  respective  angles  of 
the  key  light  and  camera.  Backlight  elevation  angle  is  set 
to  7i/4  as  recommended  by  Millerson  (Millerson,  1991). 


2  Key  light  is  the  main  source  of  light  that  establishes  the 
direction  and  the  shadows 


4.2  Selecting  Colors 

The  interaction  between  colors  assigned  for  lights 
lighting  each  area  in  a  scene  composes  the  contrast  and 
feeling  of  the  entire  scene.  Thus,  I  differentiate  between 
the  three  types  of  areas:  background,  focus,  and  non¬ 
focus.  I  calculate  contrast  and  depth  according  to  the 
difference  between  colors  assigned  to  each  of  these  areas. 

Using  the  ideal  values  for  visibility,  depth,  saturation, 
warmth,  and  lightness  for  each  area,  and  their  associated 
costs,  the  system  uses  constrained  nonlinear  optimization 
to  select  a  color  for  each  individual  light  in  the  scene  that 
will  minimize  the  cost  function: 

p 

cost (t,  c(. ,  CM )  =  P(t,  C. ,  c;_, )  +  log(— g,.  (*)),  ( 1 .4) 

j 

where  g:  R3— >-[0,1]  describes  the  desired  color  palette.  P 
is  defined  as  follows: 

As(S(ci)-s)2+Al(L(ci)-l)2  + 
P(t,ci,ci_l)=  Ad(D(Ci)-d)2  +Aw(W(Ci)-w)2  +  (1.5) 

4  (<4  (c>  ci )  -  cf  +  KhE(ci  >  ct- 1  )> 

where  S(cf)  is  saturation,  L (c*)  is  lightness  for  color  ct  and 
are  all  calculated  using  formulae  defined  in  (Castleman, 
1996).  C^(c,  Ci)  denotes  contrast  of  color  cf  given  c.  The 
values  for  the  costs  As,  Ac. ,  A^,  Ai ,  Ach,  Ac  are  given  by  the 
designer,  and  As,  Ac. ,  A^,  Ai  are  computed  by  ELE  for  every 
area  type  (focus,  non- focus,  and  background). 

5.  IMPLEMENTATION 

ELE  has  been  implemented  in  C#.  It  is  an  API  library.  It 
sits  on  top  of  a  rendering  engine.  It  has  also  been 
configured  to  use  XML  to  interact  with  rendering  engines. 
This  allowed  the  architecture  to  be  more  extensible.  We 
have  implemented  two  interfaces  for  ELE  for: 
wildtangent,  a  publicly  available  graphics  engine  used  to 
develop  web-based  games,  and  Unreal  2.0  engine  used  for 
developing  Unreal  Tournament  2003  ( UT2003 )  and  has 
also  been  used  to  develop  other  games,  e.g.  Thief  III,  and 
training  applications,  e.g.  leader’s  project  at  Institute  for 
Creative  Technologies  (Gordon,  Lent,  Valsen,  Carpenter, 
&  Jhala,  2004). 

One  concern  was  the  performance  of  ELE  within  a 
fast  paced  interactive  environment.  I  found  no 
performance  problems  when  interfaced  with  UT2003. 
ELE  ran  at  a  30  frames/second  rate,  and  thus  was 
successfully  ported  to  Unreal  platform.  There  were  some 
other  issues  concerning  the  limitation  of  the  unreal 
rendering  engine  to  render  dynamic  spot  lights,  which 
forced  us  to  use  point  lights  instead. 

During  the  implementation  and  testing  stage,  I  found 
several  benefits  of  using  ELE  over  current  methods. 
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Using  ELE  designers  were  able  to  quickly  set  the  lighting 
design  without  worrying  about  low-level  details  of 
placements,  angles,  or  color.  This  expedited  the 
development  process.  Figure  2  shows  three  different 
levels  within  UT2003  lit  using  ELE.  This  did  not  require 
any  designer  interference,  except  for  initial  setting  of 
constraints  specifying  priorities  for  the  lighting  design 
goals.  As  shown,  figure  2  illustrates  different  renderings 
where  ELE  was  configured  differently  in  regards  to 
contrast  and  overall  atmosphere.  The  upper  left  image 
shows  a  scene  where  ELE  was  configured  for  good 
visibility,  while  the  lower  right  image  shows  a  scene 
where  ELE  was  configured  for  mood  lighting,  and  thus 
the  character  in  the  lower  right  scene  was  backlit.  The  two 
right  scenes  are  set  for  a  low-key  torch  lighting,  while  the 
image  of  the  scene  in  the  lower  left  shows  a  more 
cinematic  lighting  style  with  shades  of  green. 


Fig.  2.  Different  Levels  lit  using  ELE 


6.  RESULTS 


set  by  ELE  rather  than  set  by  a  lighting  designer.  We 
invited  26  subjects  to  play  Unreal  Tournament  2003.  We 
asked  them  to  wear  a  baseball  cap  that  has  an  eye  tracker 
mounted  on  it.  We  video  taped  their  interaction.  Only  7  of 
the  subjects  have  played  Unreal  Tournament  before.  All 
subjects  have  played  non  first  person  shooter  video  games 
before. 

ELE  was  configured  with  the  following  visual  design 
goals  priorities:  visibility^  high,  visual  focus=high,  visual 
continuity=high,  others  are  all  low.  Screenshots  of  two 
rooms  configured  by  ELE  is  shown  in  figure  3. 

By  qualitatively  analyzing  the  observed  and  video 
taped  responses,  we  deduced  that  naive  users  were  able  to 
quickly  spot  the  characters  and  decipher  the  complex  3-D 
environment,  and  thus  were  able  to  react  in  a  timely 
manner.  This  is  a  great  enhancement  compared  to  the 
scene  rendered  without  ELE. 

This  is  especially  important  for  training  environments 
where  trainees  are  asked  to  accomplish  a  task  that 
involves  visual  search.  By  avoiding  the  visual  spotting 
and  complex  environment  deciphering  problem,  the 
designers  and  developers  of  training  experiences  can 
focus  on  the  actual  training  scenario. 


Fig.  3.  Visual  Focus 


6.1  Scene  Perception  and  Task  performance 

As  you  can  see  from  the  formulas  discussed  above, 
ELE  finds  the  best  lighting  setup  (layout,  angles,  and 
colors)  that  provides  good  visibility  for  action,  and  depth 
depending  on  set  priorities  for  these  goals.  Also,  as  it  can 
be  seen  by  the  algorithms  used  to  divide  the  visible  areas 
in  the  scene,  allocate  lights,  and  set  colors,  ELE  focuses 
on  configuring  lights  to  provide  best  visual  focus  (or 
visual  attention)  to  adequately  direct  user’s  attention.  The 
integration  of  all  these  methods  and  goals  enhance  scene 
perception  and  improve  users’  perception  of  complex  3-D 
environments. 

To  measure  this  quality,  we  repeated  the  eye-tracker 
experiment  described  above  with  the  same  game  ( Unreal 
Tournament  2003 )  and  level,  but  where  the  lighting  was 


6.2  Increasing  Emotional  Involvement 

Emotional  engagement  has  been  identified  to  be  of 
particular  significance  to  training  simulations,  because  of 
its  influence  on  presence  (Freeman  et  al.,  2000;  Riva  et 
al.,  2003;  Singer  &  Witmer,  1999;  Zimmons,  2004)  and 
memory  retention  (Ulate,  2002).  Many  game  designers 
use  sounds  and  visual  patterns  to  emphasis  dramatic 
moments  in  the  game  and  thus  escalate  tension  and 
increase  emotional  involvement.  For  example,  designers 
have  used  a  red  blinking  screen  to  signify  danger  (or 
health  dropping  beyond  safe  levels).  In  addition,  some 
games  designers  increase  color  saturation  between 
missions  ( Devil  May  Cry).  These  methods  are  often 
obtrusive,  and  not  continuous. 

In  contrast,  ELE  provides  a  continuous  technique  for 
increasing  tension  while  balancing  other  visual  properties. 


As  a  testing  case,  I  have  configured  ELE  to  increase 
tension  every  2  milliseconds  in  the  game,  thus,  producing 
results  shown  in  Figure  4.  As  shown  in  the  figure,  ELE 
varies  the  visual  tension  level  portrayed  while 
maintaining  visual  continuity  and  general  color  style  of 
the  scene,  which  is  a  torch  lit  scene  in  this  case.  I  have  not 
yet  validated  the  technique’s  perceptual  effect  or 
interaction  impact.  However,  the  technique  is  comparable 
to  other  techniques  used  in  cinematographers  and 
animators  (Alton,  1995;  Block,  2001;  Bucklan,  1998). 


Fig.  4.  Visual  Tension  in  the  Scene 


and  thus  accounts  and  enhances  visual  attention, 
emotional  engagement,  scene  perception  and 
understanding  in  interactive  3-D  environments.  The 
results  discussed  show  great  promise  for  the  success  of 
the  approach.  I  also  discussed  the  impact  and  utility  of 
ELE  for  interactive  3-D  training  applications. 

This  research  and  the  success  of  the  approach  trigger 
many  future  directions.  One  direction  is  to  interface  the 
lighting  system  with  several  training  applications  and 
conduct  experiments  to  evaluate  the  impact  of  the  lighting 
model  discussed  here  on  memory  retention,  learning,  and 
task  performance.  It  would  also  be  interesting  to  evaluate 
the  different  configuration  of  lighting  style  that  can  be 
achieved  using  ELE  and  measure  their  effect  on  learning 
and  training. 
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